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Resolving a -Mixture of Strategies in 
Spatial Visualization Tasks 

Abstract 

The models of standard test theory, having evolved under a 
trait- oriented psychology, do not reflect the knowledge structures 
ard the problem- solving strategies now seen as central to 
understanding performance and learning. In some applications, 
however, key qualitative distinctions among persons as to 
structures and strategies can be expressed through mixtures of 
test theory models, drawing upon substantive theory to delineate 
the components of the mixture. This approach is illustrated with 
response latencies to spatial visualization tasks that can be 
solved by mental rotation or by a nonspatial rule -based strategy. 
It is assumed that a subject employs the same strategy on all 
tasks, but the possibility of extending the approach to strategy- 
switching is discussed. 

Key words: EM algorithm, item- solving strategies, mental 

rotation, mixture models, spatial visualization. 
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Introduction 

Recent research in cognitive and educational psychology 
reveals the central role of strategies, mental models, and 
knowledge structures in learning and problem- solving. Clancy 
(1986) describes the shift in perspective to "describing mental 
processes, rather than quantifying performance with respect to 
stimulus variables." To this task, standard test theory, with its 
focus on overall level of proficiency, is ill suited. 

Consider as an example a test in which different persons 
employ different solution strategies. By characterizing examinees 
simply in terms of their propensities to make correct responses, 
the models of standard test theory (item response theory [IRT] as 
well as classical test theory) cloud analyses in several ways: 

information about subjects' mental processes is obscured, 
relationships between tasks' features and difficulties are 

confounded with strategy choice, and comparisons of subjects in 

terms of overall proficiencies are equivocal. 

Mislevy and Verhelst's (in press) mixture model approach to 

test theory can be employed to handle certain distinctions of this 

type, as when different subjects employ different solution 

strategies. The following assumptions are made: 

1. Potential strategies can be delineated a priori. 

2 . An examinee uses the same strategy on all tasks . 
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3. An examinee's iresponses are observed directly but strategy 
choice is not. 

4. The responses of examinees following a given strategy conform 
to a response model of a known form, possibly characterized 
by unknown parameters . 

5. For each strategy, psychological or substantive theory 
delineates relationships between the observable features and 
the difficulties of tasks. 

The present paper illustrates this approach by modelling 
response latencies for spatial rotation tasks that can be solved 
by either a rotational or a rule-based strategy. Simplified 
versions of processing models from the visualization literature 
are the basis of inference about strategy usage. An empirical 
Bayesian approach provides maximum likelihood estimates of the 
"structural" parameters of the problem and, for each subject, 

posterior probabilities of membership in each strategy class and 

conditional estimates of proficiency under each. 

The Data 

The data were gathered with a computer administered test of 
what are typically called "mental rotation tasks." This area was 
selected by virtue of its long history of research in both 
psychometrics (e.g., Michael, Guilford, Fruchter. & Zimmerman, 
1957; Thurstone, 1938) and cognitive psychology (e.g.. Just 
Carpenter, 1985; Lohman. Pellegrino, Alderton, & Regian, 1987). 
Of particular interest is the finding that tasks of some tests 
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designed to measure spatial visualization abilities can be solved 
with nonspatial, analytic strategies (French, 1965; Kyllonen, 
Lohman, & Snow, 1984; Pellegrino, Mumaw, & Shute, 1985). 

The tasks addressed in this paper concern a right-angled 
triangle whose vertical height was 150 units on a computer 
presentation screen, and whose horizontal side adjacent to the 
right angle was 80, 100, 120, or 140 units (see Figure 1). The 
model triangle was presented on the top half of the screen. 
Immediately below it was a second triangle whose sides were 
identical in size, but which had been rotated from the vertical by 
40, 80, 120, or 160 degrees. This target was either an exact 
match to the original or a mirror- image , and the subject was 
instructed to indicate whether the two triangles were the same or 
different. Response latency and correctness were recorded. The 
stimulus set was constructed from the Cartesian product of four 
factors: side lengths (4), rotations (4), identical or mirror- 
image (2), and hypotenuse of the model triangle left or right of 
the vertical (2), for 64 distinct stimuli in total. Because 
systematic differences are found routinely between patterns on 
rotations tasks whose correct answer is "same" compared to thosa 
for which the correct answer is "different" (e.g.. Cooper & 
Podgomy, 1976), we address only the 32 stimuli in which the 
target triangle is identical to the model. 

[Figure 1 about here] 

Subjects were male recruits to the British Army, between 18 
and 24 years of age, undergoing selection and allocation in the 
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Army Personnel Selection Centre. They were tested throughout by a 
single presenter during the week of October 17, 1988. They were 
assigned in groups of approximately 40 subjects by daily cohort of 
recruits to one of three experimental groups. All groups were 
presented two replications of the entire task set, but they 
received different instructions. Two groups started under the 
"standard testing condition" described below, and the third group 
was additionally shown a nonspatial rule-based strategy for 
solving the tasks. One of the two standard groups was given the 
rule-based strategy instruction before their second replication, 
but because only data from the first replication is addressed 
here, the two standard groups will not be distinguished. 

The standard testing condition was derived from the customary 
way of demonstrating this class of rotational tasks to naive 
subjects. An overhead projector was used to display two cardboard 
right triangles vertically, as in Figure 1, creating solid black 
images on a white background. The bottom triangle was rotated to 
several positions to show that it could be "shuffled around on the 
page" to match the top one. It was physically lifted to show that 
it could be fitted exactly over the top one. By similar means, 
the mirror image of the top triangle was shown to be different 
from the top one, no matter how it was moved about in the plane. 
The same demonstration was given for both the hypotenuse -left and 
hypotenuse -right versions of the original stimulus. This 
introduction was devised to encourage the subjects to use a mental 
rotation strategy fcr problem solution, a strategy described by 
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Shepard and Meltzer (1971) as "isomorphic." Under this strategy, 
degree of rotation is a primary determiner of task difficulty, in 
terms of both response latency (Shepard & Meltzer, 1971) and 
accuracy (Tapley & Bryden, 1977). 

The second experimental condition was devised to encourage 
the use of an "analytic" or "rule-based" strategy. Such 
strategies employ procedural learning rules rather than mental 
visualization to determine correct answers. Subjects were shown 
how to judge whether the target was identical or mirror- image of 
the model by attending to a specific feature of the triangle, 
namely the length of the side clockwise adjacent to the right 
angle. The triangles are different if one encounters the long 
side of one triangle but the short side of the other. It can be 
hypothesized that under this strategy, difficulty is nearly 
independent of the degree of rotation, but depends primarily on 
the saliency of the key feature; that is, the strategy should be 
more difficult to implement as the righc triangle becomes more 
nearly isosceles. 

The rule was derived by asking British Naval Engineering 
cadets at the Royal Naval Engineering College at Plymouth how they 
habitually solved such tasks. While some replied they knew just 
by looking at them (i.e., their strategies were not consciously 
available), others said they never rotated the objects themselves; 
they instead moved their gaze aroxmd the model from a fixed point 
on it, noting the presence or absence of some salient feature. 
They then found the same starting point on the target and checked 



for the presence or absence of the feature. If it was there, the 
target had to ba the same as the model. If not, it had to be 
different . 

Data from 244 subjects were initially made available for this 
study. The analyses reported here concern the 196 remaining after 
trimming those with the highest and the lowest 10-percent of 
within task- type variance, as pooled over the 16 rotation/side - 
length pairs, so as to leave the remaining 80-percent nearly 
homoscedastic with respect to this component of variation. Of 
these, 131 had not been instructed in the rule and 65 had. Note 
that the 2:1 ratio reflects equal trimming of noninstructed and 
instructed subjects. 

Figures 2 and 3 plot median latencies to "same" tasks against 
coded angular displacement and side length, respectively. 
Throughout the paper we work with natural logarithms of response 
times, standardized over all tasks and subjects. Figure 2 shows 
that average response latency tends to increase linearly with 
angular displacement, which is coded as -1.5, -.5, .5, and 1.5 
along the x-axis. The plot symbols are coded side lengths, from 
shortest (most acute) to longest (most nearly isosceles) as -2.25, 
-1.25, .75, and 2.75. Coding side length in these unequally 
spaced intervals produces the nearly linear relationship shown in 
Figure 3, where the x-axis gives coded side length and plot 
symbols give coded angular displacements. Table 1 gives three 
representative subjects' observed means, averaging over left- and 
right-hypotenuse tasks within rotation and side- length. 



[Figures 2 and 3 about here] 
[Table 1 about here] 

The Model 

A mixture model for the data described above is laid out in 
two phases. First is the response model . which concerns the 
distribution of response latencies conditional on choice of 
strategy and proficiency in using that strategy. Second is the 
population model . which concerns the proportions of subjects that 
employ the various strategies and the distributions of proficiency 
within strategy classes. The generic model described below is for 
a mixture of K strategy groups; a single strategy can be modelled 
by setting K to one. After presenting the model, we give the 
results of fitting three single -strategy models and a mixture of 
two strategies. 

Tlie Response Model 

Suppose that K potential strategies are available to solve 
tasks in a test, and each subject uses one strategy exclusively on 
all tasks. Each subject will be characterized by two vector- 
valued unobservable variables. The first is the indicator 
variable 4 - ((f>^, . . . ,4>^) , where (f>^ takes the value 1 if Strategy k 
is the one the subject uses, 0 if not. The second is 
£ - (.8^, . . , ,d^) , where 6^ is proficiency under Strategy k. Only 
one of the elements of 6 plays a role in producing the data 
observed frou a given subject, namely the one for which 
Task j is characterized by the vector of difficulty parameters 
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~ (^ji '^jK^' determines the difficulty of Task j 

for subjects who employ Strategy k. 

We work with a log normal distribution for the response 
latency t^. of a subject to Task j, given that Strategy k is 
employed. Defining x as in t and 5 as a scale parameter of the 
log-normal density pertaining to Strategy k, the response model is 
given by 



f(x.|^.£.^..£) -n f^(x.|.j^.^.j^..^) 



[1] 



where 



V-ji\'^jk'V 



exp 



n 



• [2] 



Thus [2] gives the density for log response time to Task j for a 
subject employing Strategy k, with proficiency under that 
strategy. The difference is the mean and is the 

standard aviation. Note that lower values of & signify faster 
expected response times (i.e., more proficiency on the part of the 
subject), while higher values imply slower response times. Lower 
values of /3 signify slower expected response times (greater 
difficulty on the part of the task) , while higher values imply 
faster response times. The product over k that appears in [1] 
serves merely to select the f^^ that applies, since is 1 in this 
case and 0 in all others. In the sequel f, (x.|5 ,^ ,5 ) will be 
abbreviated at times as fj^(Xj|$j^), suppressing the dependence on 

^jk K- 



Equations [1] and [2] posit individual differences among 
subjects as to strategy selection and speed-within- strategy . 
Individual differences corresponding to slopes might also be 
entertained, but to do so here increases complexity without adding 
insight into the inferential approach. We note in passing, 
however, that recent studies suggest that at least some of the 
slope differences among individuals found in standard analyses may 
be related to the use of different strategies (Carter, Pazak, & 
Kail, 1983; Just & Carpenter, 1985). 

Psychological theory about what makes tasks easy or hard 
under various strategies appears in the form of models for the 
ySj^'s. Following Scheiblechner (1972) and Fischer (1973), let q^^ 
be a known vector of coefficients expressing the extent to which 
Task j exhibits each of Mj^ features that determine task difficulty 
under Strategy k; let be a parameter vector, also of length Mj^, 
that conveys the influence of those features . Assuming a linear 
relationship between task features and task difficulty, 

\ 

^jk - ^ ^jkm °km - 3jk Hk • 
•' m-1 

For the rotational strategy, for example, angular 
displacement is a major determiner of difficulty. Tasks rotated 
to the four equally- spaced displacements will have q values for 
this feature coded as -1.5, -.5, .5, or 1.5. Tlie corresponding 
element of a would indicate the incremental difficulty associated 
with an increase of one additional unit of angular displacement. 
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- a strategy under which side length is posited to contribute to 
difficulty, tasks will have q values of -2.25, -1.25, .75, or 2.75 
that indicate the saliency of this key feature. The corresponding 
element of a would reflect the additional difficulty resulting 
from an additional increment toward being isosceles. 

Task responses are assumed to be independent given a, 5 and 
t. Letting x-(x^ x^) be a vector of log latencies to n tasks. 



so that 



^(5l£'^'2'£) - n f, (x|5, ) ^ 



^ k^i'-'k^ ■ [4] 



The Population Model 

Suppose that subjects are a representative sample from a 

population in which the proportion employing Strategy k is 

With 0<.j^<l. Denote by . the vector (.^ . Denote by ^ 

gj^C^l^) the density function of 6^ for members of Class k. We 

assunie normal distributions (although other distributions or even 

nonparametric approximations could be used), so that g^(. |^ ) 

k Ic Ic 



has the form 



72 ^''k-V^ 

2a, ^ 
k 



For brevity, denote the population distribution parameters by r 



Estimation 

Equation [4] is the conditional probability of a response 
pattern x, or the probability of observing x from a subject having 
particular values of 9 and <l>. Assuming the population model 
described above, the probability of observing x from an examinee 
selected at random, or the marginal probability of x, is given as 

k 

Let X - (x^,...,Xj^) be the response matrix of a random sample of N 
subjects to n tasks. A realization of x induces the marginal 
likelihood function for a, S, ?r, and T as the product over 
subjects of factors like [5]: 

N 

L(a,5,7r,r|x) - n p(x |a,5,7r,r) . [6] 
i_l 

We refer to q, 6, »r, and F as the structural parameters of the 
problem. Their number remains constant as N increases. The 
incidental parameters 6 and <f>, whose numbers increase in direct 
proportion to N, have been eliminated by marginalizing over their 
respective distributions. Marginal maximum likelihood (MML) 
estimation finds the values of the structural parameters that 

AAA A 

maximize [6], say a, 5, w, and F. The Appendix gives a numerical 
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solution for the present problem using Dempster, Laird, and 
Rubin's (1977) EM algorithm. The solution is for data in which 
strategy use is not known with certainty for any subject, as was 
the case in the present study. If strategy-use info- ;mation is 
available for some subjects, however, as from follow-up interviews 
with a subsample, estimates of structural parameters can be 
improved- -sometimes dramatically- -by exploiting it (Titterington, 
Smith, & Makov, 1985, Section 4.2). 

Once MML estimates of structural parameters have been 
obtained, one can obtain empirical Bayesian approximations of 
probabilities of class membership for any examinee, and estimates 
of condit-.onal on membership in any class k. If the structural 
parameters were known with certainty, the posterior density for 
Subject i would be 

P^\'^k"-'-'-i'2'£'!I'r) 

'^k ^k<5il^.2k'V ek^^lHk^ 



[7] 



An empirical Bayes approximation substitutes MLEs for the 
structural parameters. The posterior probability that Subject i 
employed Strategy k, denoted P^^^, is approximated as 



A A A A 



^ik " ^^^k"^l5i) " S P(^-V^l5i.«.£.2.r) de^ . [8] 

Conditional on membership in Class k, the posterior expectation 
and variance of 6^ are approximated as 

12-1 O 
J- ^ 



-1.x.) « S P(^.^i,-i|x,.a.i.2.r 



) [9] 



and 



Var(^^|^.^-l.x.) » 



[10] 



Single -Strategy Solutions 



This section uses single -strategy solutions to illustrate Q 
matrices, obtain baseline likelihood statistics, calculate task- 
difficulty estimates v ider different models,' and give a feel for 
the data by looking more closely at the subjects introduced in 
Table 1. These solutions assume not only that a subject applies 
the same strategy on all tasks, but that all subjects are applying 
the same strategy. Three such solutions are considered: 

1. Rule-based (difficulty depends on side length only), 

2. Mental rotation without side- length effects (difficulty 
depends on degree of rotation only) , and 

3. Mental rotation with side-length effects (difficulty depends 
on both side length and degree of rotation) . 

Side -Length Only 

Suppose that all subjects followed a rule -based strategy, 
under which difficulty is determined by side length alone. Under 
this scenario, the only systematic sources of variation in 
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response times are subjects' overall proficiencies and tasks' 
differing side lengths. Proficiency is denoted by the univariate 
variable 6, assumed to follow a normal distribution whose mean ^ 
and standard deviation a are to be estimated. Task difficulty is 
given as /3j - q^^ a^, where q^.^ is the coued side- length of task j 
(-2.25, -1.25, .75, or 2.75), and is the change in difficulty 
associated with an additional increment in side length toward 
isosceles. Finally, 5 is the standard deviation of log response 
time within task type and subject. 

The first panel of Table 2 gives the MLEs of the structural 
parameters. The value of -2 log likelihood, or -2 in 2 p(x^), 
also appears, which will be used to compare the fit of alternative 
models. The estimate of is -.124, which translates into 
expected increases in response latencj as side length increases. 
The first panel of Table 3 gives the resulting /9s, with higher 
values corresponding to faster response times and lower numbers 
corresponding to slower ones. The modelled standardized log 
latency of a particular subject on a particular task type under 
this model would be obtained by subtracting the task's p value 
from that subject's e value- -thereby maintaining for all subjects 
the pattern of increasing difficulty with increasing side length 
without effects of degree of rotation. 

(Insert Tables 2 and 3 about here] 
This pattern is illustrated with the subject whose observed 
means data appear in the second panel of Table 1. The Bayes 
estimate 6 is -.317, a measure of overall proficiency. Combining 
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9 with the gives the modelled latencies in the second panel of 
Table 4. The third panel gives residuals, which indicate faster 
averages than expected in some cells, slower in others, but no 
systematic trend in average residuals in either the side- length or 
rotation margins. (The residuals do not sum to zero because the 
expectations were based on a Bayesian proficiency estimate, as 
opposed to, say, least-squares within subject.) Table 5 gives 
residuals for all three sample subjects. Note the strong trend 
related to rotation for the last subject. 

[Insert Tables 4 and 5 about here] 

Rotation Only 

Even though the side- length model fits the data of subjects 
like the second fairly well, the familiar relationship between 
difficulty and degree of rotation appears in the residuals of 
subjects like the fifth. An alternative simple model has 
difficulty depend on rotation only, neglecting the possible impact 
of differences in side-length. Under this model, - q^^ a^, 
where q^^ is the coded rotation of Task j (-1.5, -.5, .5, or 1.5) 
and is the impact on difficulty of an additional ■'ncrement in 
rotation. The results of fitting this model appear in the second 
panel of Table 2. The MLE of is -.108, signalling increasing 
difficulty as the degree of rotation increases. The resulting y9s 
are in the second panel of Table 3, showing patterns of expected 
difficulty that depend on rotation alone. 

The value of -2 log likelihood is 14907.78. To compare the 
fit of alternative, possibly nonnested, models to the same data 
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set, Akaike (1985) calculates the index AIC ("an information 
criterion") for each model: -2 log likelihood plus twice the 
number of parameters estimated. The model with the smaller nlC is 
preferred. AIC for rotation-only is 14907.78 + 2x4 - 14915.78. 
AIC for side-length only, which also has four parameters, is 
14432 .40- -smaller by 483. The side-length only model fits better 
than the rotation only model. 

Subject 6 estimates in this model combine with the ^s to give 
expected response latencies that maintain the same relative 
relationships seen in the ^s , but vary as to overall speed. 
Subtracting such expectations calculated with 6s from the observed 
data gives the residuals shown in Table 6 for the sample subjects. 
The third subject's strong linear trend in residuals associated 
with rotation has been dampened considerably, but a reverse trend 
in the rotation margin has been introduced for the first two 
subjects. Note also the tendency toward a consistent ordering in 
the side -length residuals. 

[Table 6 about here] 

Rotation and Side-Length 

A final single-strategy model incorporates main effects for 
both rotation and side- length: " ^jl °l "2' 
and as have the same meanings as in the preceding models . This 
incorporates a more elaborate scenario for mental rotation, 
allowing for latency to increase not only with degree of rotation, 
but with the difficulty of comparison after rotation is complete 
(Cooper & Podgorny, 1976). Parameter estimates and -2 log 
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likelihood appear in the third panel of Table 2. Because of the 
orthogonal experimental design, as are identical to those of the 
previous single -feature models. AIC for this model is 14258.72, 
smaller than the side-length AIC by 173 and smaller than the 
rotation AIC by 661. The model incorporating both task features 
thus fits better than both models with a single feature only, even 
accounting for the additional parameters being estimated. The 
resulting /3s appear in the third panel of Table 3. 

The residuals for the sample subjects are in Table 7. 
Because of the orthogonal design, the rotation margins of the 
residual tables under this model match the corresponding margins 
under the rotation only model, and the side-length margins match 
those from the side-length only model. 

[Table 7 about here] 

A Two- Strategy Solution 
This model posits a mixture of two types of subjects: those 
employing the rule -based strategy, whose response patterns can be 
largely captured with the side-length only model, and those 
employing mental rotation, whose responses depend on both side- 
length and degree of rotation. There are ten parameters to 
estimate in this model: the relative proportion of strategy use in 
the sample; a mean, standard deviation, and within task- type 
standard deviation for each strategy; an a for side -length for the 
rule-based strategy component; and two as, one for side-length and 
one for rotation, for the rotation strategy component. The 
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estimates appear in the final panel of Table 2. AIC improves over 
the rotation and side-length single -strategy model by 72. 

Two distinct sets of )9s result in this model, one for each 
strategy. These appear in the last two panels of Table 3. The 
side-only component, like the side-length only single strategy 
panel, shows only a side-length effect. Interestingly, it is 
smaller than the corresponding effect in the side-length single- 
strategy model. The "side -length and rotation" component shows 
both effects. Compared to the corresponding single strategy 
solutions, both the side -length and the rotational effects are 
stronger. The interpretation would be that decreasing the 
distinctness of the salient feature of the stimulus hampers 
subjects employing rotation more than those employing the rule. 

The point of mixture modelling is that wft do not know with 
certainty which subjects are employing which strategy. One of the 
structural parameters is the proportion using each; MLEs are 57- 
percent for the rule-based strategy and 43-peicent for the 
rotational strategy. These values can be expressed as the 
averages of subjects' posterior probabilities of being in one 
strategy group or the other (Equation 8). The histograms in 
Figure 4 show subjects' posterior probabilities for the rule- 
based strategy. Most have probabilities below .2 or above .8, 
indicating fairly good separation of the components of the 
mixture. To put this in the context of the statistical literature 
on mixtures (e.g., Titterington et al., 1985), Figure 5 gives the 
histogram of a closely matching "mixture of homoscedastic Gaussian 
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components" problem. The mixing probability is .55 and the 
distance between the means of the components is 2.2 standard 
deviations. The information about the mixing proportion is about 
half what it would be if component membership were observable 
(Hill, 1963). Figure 6 breaks the information from Figure 4 down 
by whether subjects were shown the rule. The proportion of 
instructed subjects whose responses are strongly allied with the 
rule-based strategy is substantially higher- -nearly half, as 
compared to just a fourth of those not instructed. 

[Insert Figures 4-6 about here] 

Table 8 gives posterior probabilities and residuals for the 
sample subjects. As suggested by patterns of residuals from the 
single strategy models, the posterior probabilities of Subjects 1 
and 2 are concentrated on Strategy 1, the rule-based strategy (Ps 
of .98 and .70), and that of Subject 3 is concentrated on the 
rotational strategy (P nearly 1.00). The main factor that 
determines strategy assignment is the presence of a linear trend 
related to rotation. Of secondary importance is the strength of 
the trend related to side length, with stronger trend being 
associated with the rotational strategy. The high probability of 
the rule -based strategy for Subject 1 is a consequence of both a 
lack of trend for rotation and a weak trend for side length in the 
observed da :a (see Table 1) . 

For a decision-making problem that depends on identifying a 
subject's strategy use, the strategy with the higher posterior 
probability is the choice. For a problem that depends on 
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predicting future performance, the optimal prediction is a 
mixture. For Subject 2, for example, predictions would be 
calculated under both strategies, using the task parameters and 
his posterior d distribution within each; these distinct 
predic'_ions would be averaged with weights of .70 for the rule- 
based strategy and .30 for the rotational one. The predictions 
for the residuals in Table 8 were calculated in this manner, using 
conditional within-strategy posterior means for ds. 

[Insert Table 8 about here] 

Discussion 

This presentation was meant to demonstrate how a mixture - 
model approach can be incorporated into test theory to deal v?ith 
different problem .solving approaches. We readily concede that the 
psychological model for the visualization tasks is overly 
simplistic. There are some ways it could be made more realistic 
while preserving the assumption that a subject maintains the same 
strategy throughout observation. These include modelling 
subprocesses , relaxing distributional assuanptionL , or 
incorporating individual difference terms for residual variances 
or sensitivity to rotational angles. But more substantial steps 
toward reality lie beyond this framework. 

It must first be admitted that the two strategies discussed 
here do not exhaust the variety of approaches that subjects bring 
to bear upon such tasks. They may be viewed as archetypes, one or 
the other of which may be sufficiently close to a given subject's 
data to provide a serviceable guide to selection or instruction. 
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In applied work, analyses of subjects' fit and patterns of 
residuals will be important to flag patte rns that are not 
modelled, either for special consideration if they are few in 
number, or for incorporation into the model if they are recurrent. 

Perhaps more importantly, we must consider the possibility of 
strategy switching, or the violation of the assumption that a 
subject follows the same strategy on all tasks. This assumption 
may be reasonable if strategy is defined through knowledge 
structures, so that response patterns associated with advanced 
structures are not accessible to learners in earlier stages. An 
example of this type is Siegier's -nalysis (1981) balance beam 
tasks , where competence is increased by adding rules to a 
repertoire in a largely predictable order. It may also be 
reasonable if strategies associated with alternative mental models 
of a domain are unlikely to coexist in a given subject. 

The same assxamption is less reasonable in a domain where 
alternative strategies are available to the same subject, so that 
the conditions and the frequency of strategy switching become 
additional sources of individual difference. Kyllonen, Lohman, 
and Snow (1984) note increasing use of non- spatial strategies as 
mental rotation tasks become more difficult. A more appropriate 
model in this case would characterize mixtures within subjects; 
not jus't that a subject followed Strategy A or Strategy B, but 
that she used Strategy A on 30 -percent of the tasks and Strategy B 
on 70-percent- -and that the characteristics of tasks that were 
relevant to her strategy choice were such-and-such. Given the 



vicissitudes of even the straight mixture model, it is clear that 
modelling mixtures within subjects will require richer information 
than the familiar correctness or response times, perhaps in the 
form of response protocols, intermediate products, or physical 
measures such as eye movement (as in Just & Carpenter, 1976). 

It would appear that mixture models of the type illustrated 
here have most promise in contexts where the qualitative 
distinctions among persons are relatively few in number, stable 
during the period of obsexrvation, and distinguishable in terms of 
their implications for obsexrvable behavior. We have demonstrated 
by example that the calculations in this case are tractable and 
the mixtures can be resolved satisfactorily. Future work will 
focus on a more general computational scheme and a broader variety 
of more educationally relevant applications . 
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Appendix: Estimating Structural Parameters via the EM Algorithm 

Equation [5] is an "incomplete data" density function of the 
form addressed by Dempster, Laird, and Rubin (1977) in "Maximum 
likelihood from incomplete data via the EM algorithm." Estimating 
the structural parameters would be straightforward if values of 
the latent variables 9 and ^ could be observed from each subject 
along with his or her response vector x; this would be a "complete 
data" problem. The EM algorithm maximizes the incomplete- data 
likelihood [6] iteratively. The E-step, or expectation step, of 
each cycle, calculates the expectations of the summary statistics 
that the complete-data pioblem would require, conditional on the 
observed data and provisional estimates of the structural 
parameters. The M-step, or maximization step, solves what looks 
like a complete-data maximum likelihood problem using the 
conditional expectations of summary statistics. The resulting 
maxima for the structural parameters are improved estimates of the 
incomplete -data solution, and serve as input to the next E-step. 

We employ the variation of the EM algorithm suggested by 
Mislevy and Verhelst (in press) to estimate the parameters of 
mixtures of psychometric models. The integration that appears in 
[5] is approximated by summation over a fixed grid of points. The 
E-step calculates, for each examinee, conditional probabilities of 
belonging to each component of the mixture (i.e., strategy class) 
and, conditional on component membership, the probabilities that 6 
takes various grid-point values. The grid points play the role of 
weighted pseudo-data points in the M-step. Numerical integration 

26 O 9 



could be avoided in the special case considered above, but the 
approach described below also applies with alternative response 
models and distributional forms. 



Solving the "Complete Data" Problem 
This section gives the ML solution for a, 6, k, and F that 
would obtain if values of S and <f> were observed for each subject 
along with x. Among the sampled subjects from Strategy Class 
k, some niomber I^<N^ distinct values of 6 would be obseirved, say 
9^^ for i-l,...,I^. Define the following statistics: 



^iki' indicator variable that ':akes the value 1 if Subject 
i is in Strategy Class k and has proficiency 6^^^ • 



o N, , the niomber of examiniees obseirved to be in Class k: 
k 



\ - f ^ik - ^ ^ ^iki • [^^^ 
i 1 i 

N^^ , the number of examinees in Class k with ^"6^^^- 



\i - ^ ^iki • 1^2] 
■ 1 



X., , the mean of ^ -centered log latencies for Task j from 
J 



subjects in Class k: 



^jk-\' (-ij-V ^iki • 

2 

Sj^, the ave.rage square of ^-centered log latencies for Task 
j from subjects in Class k: 
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The complete data likelihood for o, 5, jt, and r that would be 
induced by the observation of x, fl, and 4> can be written as 

L*(a,£,Jr,r|x,£,^) - 

n p(Nj^|.) n F(Nj^,|Nj^.a) n \(^,\e^,,a^.6y'^' , 



whence the complete data log likelihood 



i*(a,£,2r,r|x,^,^) - S Nj^ log S log 



ML estimation for the complete data problem proceeds by solving 
the likelihood equations, which are obtained by setting to zero 
the first derivatives of [AS] with respect to each element of 
^-'i'^'D- "^^t is, for a generic element u of (a,5,7r,r). 



d£*(a,S,n,r\x,e,4) 

' ^T^^^^^ • ^^^^ 

9u 



For elements of n, one must impose the constraint that Sjt -1 

k ' 

say with a Lagrangian multiplier. One obtains a closed form 
solution for the proportion of subjects in each strategy class: 
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[A7] 



For elements of the population parameter vector T, the ML 
estimates in the normal case are, for k-l,...,K, 



[A8] 



and 



^k - \' 5 <\i-^k)' \i 



[A91 



For the parameters of the task response model, 



2k- - (QkQP" Qk5k ■ 

where Q^^ - [q^^ ... 3^^] and - (k^^ x^^) , and 



[AlO] 



"2 -1 r 2 " - "2 



[All] 



Solving the "Incomplete Data" Problem 
The likelihood equations in the incomplete -data problem, in 
which 6 and 4> are not observed, are the first derivatives of the 
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log of the marginal likelihood function, [6], set to zero. Under 
mild regularity conditions, these take the form of a data-weighted 
average of the complete-data likelihood equations shown as [A6] . 
Letting ^ia,S,Tr,r\x) - in L(a.£.i[r,£|x) , the incomplete -data 
likelihood equation for a generic element u of (a,6,7r,r) is 



8 2(a,S,n,r\X) 
0 - 

du 



O-Sj p(^,<^,-l|x) d8 , [A12] 

k du _ K - ~ 



where 

p(x 1^ ,<f> ,cc,S.n,r) 
p(£,j|x) - n . [^3^3j 

i P(x^|a,£,7r,r) 

It is seen in [5] that evaluating tha denominator of [Al3] 
involves integration over the li(n^,cT^) distributions. We 
approximated these integrals via Gaussian quadrature. Tabled 

values are obtained for L points e^, with associated 

weights W^,...,W^. (We used L-80, ranging from -3 to +3). 
Calculation proceeds as though these were the only possible values 
for 9; I^j^^ indicates whether Subject i used Strategy k and had 
proficiency 9^. 

In the incomplete-data problem, neither the values of 8 nor 

~i 

4^ are known, so neither are the 1. values used to calculate 



summary statistics. If the values of the structural parameters F, 
TT, S, and a were known, however, it would be possible to calculate 
the expected values of the Ij^^^s given x^s as follows: 



^iki - E(^ikii5i.2.£.^n 



V5il«i.2k.V M^^iV 

[A14] 
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In the E-step of the EM algorithm, one evaluates [Al4] for 

each i, k, and i using' provisional estimates of a, 6, tt, and F. 

One then obtains expectations of the summary statistics defined in 

"2 

[Al]-[A4], say N^, N^^ , x^^, and s^^. The grid values 9^ in the 
incomplete -data solution thus correspond to the observed f values 
in the complete data solution. 

In the M-step, one solves facsimiles of the complete data 
likelihood equations, [A7]-[A11], with N^, N^^, x^^, and s^^ in 
place of their observed counterparts. Cycles of E- and M-steps 
continue until changes are suitably small. Convergence to a local 
maximum is assured, except from initial values that lie on 
boundaries of the parameter space (e.g., tt^-O) . Repeated 
solutions from different starting values help identify the global 
maximum. Accelerating methods may be used if convergence is too 
slow. 

Equation [A14] will be recognized as an application of Bayes 

theorem, giving the posterior probability that ^j;^"®^ and ^^^^"1 

after observing x^^. The normalizing constant in the denominator 
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IS an approximation of p(x^) as given in [5]. During the E-step, ' 
one may therefore accumulate the sum -2 S log p(x.) to track the 
performance of improvement in fit over cycles, to examine the fit 
obtained with various values of structural parameters, or to 
compare the fit of alternative models. 

Empirical Bayes Estimates of Examinee Parameters 
The numerical approximations employed above to estimate 
structural parameters can be used for subsequent empirical 
Bayesian estimates for individual subjects. The expectations of 
the indicator variables I^^^ are evaluated via [A14] with MML 
estimates of a, S, tt, and T. The empirical Bayes approximation of 
probability of membership in Strategy Class k is given as 

^k- P(^k-=^l5i) -^^iU • [A15] 

Conditional on membership in Class k, the posterior expectation of 
is approximated as 



^k - ^(^k'^ik-^-^i) - ^k' ^ «i ^iki [^16] 



and the corresponding posterior variance is 



Var(..^|^^^-l.x^) . i^^^ Z e] 'l.^^ - e.l . [A17] 
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Table 1 

Examples of Observed Standardized Mean Log Response Times 



Subject 1 

Rotation 



Side Length 




40 




80 




120 




160 


Average 


80 


-0 


069 


0 


059 


-0 


508 


-0 


676 


-0 


299 


100 


-0 


724 


-0 


483 


-0 


242 


-0 


383 


-0 


458 


120 


-0 


591 


-0 


828 


0 


406 


-0 


993 


-0 


502 


140 


-0 


288 


-0 


287 


-0 


485 


-0 


483 


-0 


386 


Average 


-0 


418 


-0 


385 


-0 


207 


-0 


634 


-0 


411 


Within Cell 


Pooled 


Standard Deviation 






0 


779 



Sub] act 2 



Rotation 



Side Length 




40 




80 




120 




160 


Average 


80 


-1 


692 


-0 


277 


-1 


012 


-0 


863 


-0 


961 


100 


0 


705 


0 


514 


-0 


450 


-0 


871 


-0 


025 


120 


-1 


082 


-0 


912 


0 


224 


0 


126 


-0 


411 


140 


0 


661 


0 


015 


-0 


316 


-0 


035 


0 


081 


Average 


-0 


352 


-0 


165 


-0 


389 


-0 


411 


-0 


329 


Within Cell 


Pooled Standard Deviation 






0 


713 


Sub-ject 3 






























Rotation 












Side Length 




40 




80 




120 




160 


Average 


80 


-0 


413 


-0 


545 


-0 


192 


-0 


304 


-0 


363 


100 


-1 


699 


-1 


849 


-0 


108 


0 


223 


-0 


858 


120 


-1 


186 


-0 


T)19 


-0 


189 


0 


.966 


-0 


107 


140 


-0 


451 


-0 


815 


-0 


525 


0 


.404 


-0 


347 


Average 


-0 


937 


-0 


807 


-0 


.254 


0 


.322 


-0 


.419 



Within Cell Pooled Standard Deviation 0.687 



Table 2 

Parameter Estimates and -2 Log Likelihood 



Side Len gth Only -2 log likelihood - 14424.40 

/i o S q(1) 

0.000 0.645 0.726 -0.124 

Rotation Only -2 log likelihood - 14907.78 

P a 6 q(2) 

0.000 0.644 0.756 -0.108 

Side Length and R otation -2 log likelihood - 14248.72 

a 6 q(1) q(2) 

0.000 0.645 0.716 -0.124 -0.108 

Two Strategy Mode]. -2 log likelihood - 14167.03 

Strategy »r p a 5 q(1) tt(2) 

Rule .566 0.099 0.745 0.668 -0.104 

Rotation .434 -0.129 0.455 0.753 -0.151 -0.219 
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Table 3 
Task Difficulty Parameters 



Sid:: Length Only Model 

Rotation 



Length 




40 




80 




120 




160 


80 


0 


279 


0 


279 


0 


279 


0 


279 


100 


0 


155 


0 


155 


0 


155 


0 


155 


120 


-0 


093 


-0 


093 


-0 


093 


-0 


093 


140 


-0 


341 


-0 


341 


-0 


341 


-0 


341 



Rotation Only Model 

Rotation 



Length 




40 




80 




120 




160 


80 


0 


162 


0 


054 


-0 


054 


-0 


162 


100 


0 


162 


0 


054 


-0 


054 


-0 


162 


120 


0 


162 


0 


054 


-0 


054 


-0 


162 


140 


0 


162 


0 


054 


-0 


054 


-0 


162 



Side Length and Rotation Model 

Rotation 



Length 




40 




80 




120 




160 


80 


0 


441 


0 


333 


0 


225 


0 


117 


100 


0 


317 


0 


209 


0 


101 


-0 


007 


120 


0 


069 


-0 


039 


-0 


147 


-0 


255 


140 


-0 


180 


-0 


287 


-0 


395 


-0 


503 



(continued) 



Table 3, continued 
Task Difficulty Parameters 



Two Strategy Mode l: Side Length Only Component 

Rotation 

Side Length 40 80 120 160 

80 0.234 0.234 0.234 0.234 

100 0.130 0.130 0.130 0.130 

120 -0.078 -0.078 -0.078 -0.078 

140 -0.286 -0.286 -0.286 -0.286 

Two Strategy Mode l: Side Length and Rotation Component 

Rotation 

Side Length 40 80 120 160 

80 0.667 0.448 0.229 0.010 

100 0.517 0.298 0.079 -0.140 

120 0.215 -0.003 -0.222 -0.441 

140 -0.086 -0.305 -0.524 -0.742 



Table 4 

Modelling Subject 2 under Side Length Only 



Observed Log Response Times 

Rotation 



Side Length 




40 




80 


120 




160 


A VP facrp 


80 


.1 


692 


-0 


277 


-1.012 


-0 


863 


-0 . 961 


100 


0 


705 


0 


514 


-0.450 


-0 


871 


-0 .025 


120 


-1 


082 


-0 


912 


0.224 


0 


126 


-0.411 


140 


0 


661 


0 


015 


-0.316 


-0 


035 


0.081 


Average 


-0 


352 


-0 


165 


-0.389 


-0 


411 


-0.329 


6 - -.317 


















Modelled Log Response 


Times 


















Rotation 








Side Length 




40 




80 


120 




160 


Average 


80 


-0 


599 


-0 


599 


-0.599 


-0 


399 


-0.599 


100 


-0 


475 


-0 


475 


-0.475 


-0 


475 


-0 475 


120 


-0 


227 


-0 


227 


-0.227 


-0 


227 


-0.227 


140 


0 


021 


0 


021 


0.021 


0 


021 


0.021 


Average 


-0 


320 


-0 


320 


-0.320 


-0 


320 


-0.320 


Residuals 


























Rotation 








Side Length 




40 




80 


120 




160 


Average 


80 


-1 


096 


0 


319 


-0.416 


-0 


.267 


-0.365 


100 


1 


177 


0 


986 


0.021 


-0 


.399 


0.446 


120 


-0 


859 


-0 


689 


0.447 


0 


.350 


-0.188 


140 


0 


636 


-0 


010 


-0.341 


-0 


.060 


0.056 


Average 


-0 


035 


0 


.152 


-0.072 


-0 


.094 


-0.013 



Root Mean Square Error 0.619 



43 



Table 5 

Residuals from Side Length Only Model 



Subject 1 (5 - -.40) 

Rotation 

Side Length 40 80 120 160 Average 

80 0.605 0.733 0.167 -0.002 0.376 

100 -0.174 0.067 0.309 0.168 0.092 

120 -0.289 -0.526 0.708 -0.691 -0.199 

140 -0.234 -0.233 -0.431 -0.429 -0.332 

Average -0.023 0.011 0.188 -0.238 -0.016 

Root Mean Square Error 0.426 

Subject 2 (e ~ -.32) 

Rotation 

Side Length 40 80 120 160 Average 

80 -1.096 0.319 -0.416 -0.267 -0.365 

100 1.177 0.986 0.021 -0.399 0.446 

120 -0.859 -0.689 0.447 0.350 -0.188 

140 0.636 -0.010 -0.341 -0.060 0.056 

Average -0.035 0.152 -0.072 -0.094 -0.013 

Root Mean Square Error 0 619 

Subject 3 (e - -.40) 

Rotation 

Side Length 40 80 120 160 Average 

80 0.269 0.138 0.490 0.379 0.319 

100 -1.140 -1.291 0.450 0.781 -0.300 

120 -0.877 0.290 0.121 1.276 0.203 

140 -0.389 -0.753 -0.463 0.465 -0.285 

Average -0.534 -0.404 0.149 0.725 -0.016 

Root Mean Square Error 0 703 
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Table 6 

Residuals from Rotation Only Model 



Subiect 1 (e - - .39) 

Rotation 



Side Length 




40 




80 




120 




160 


Average 


80 


0 


487 


0 


507 


-0 


168 


-0 


444 


0.095 


100 


-0 


168 


-0 


035 


0 


098 


-0 


151 


-0.064 


120 


-0 


035 


-0 


380 


0 


746 


-0 


761 


-0.108 


140 


0 


268 


0 


162 


-0 


145 


-0 


251 


0.009 


Average 


0 


138 


0 


063 


0 


133 


-0 


402 


-0.017 



Root Mean Square Error 0.374 



Subiect 2 (e - -.32) 

Rotation 



Side Length 


40 




80 




120 




160 


Average 


80 


-1.215 


0 


093 


-0 


750 


-0 


710 


-0.646 


100 


1.183 


0 


884 


-0 


189 


-0 


717 


0.290 


120 


-0.605 


-0 


543 


0 


485 


0 


280 


-0.096 


140 


1.138 


0 


384 


-0 


055 


0 


119 


0.397 


Average 


0.125 


0 


205 


-0 


127 


-0 


257 


-0.014 


Root Mean Square Error 














0.694 


Subject 3 (B 


- -.40) 






















Rotation 








Side Length 


40 




80 




120 




160 


Average 


80 


0.150 


-0 


089 


0 


155 


-0 


064 


0.038 


100 


-1.135 


-1 


394 


0 


240 


0 


462 


-0.457 


120 


-0.623 


0 


436 


0 


159 


1 


206 


0.294 


140 


0.113 


-0 


359 


-0 


177 


0 


644 


0.055 


Average 


-0.374 


-0 


352 


0 


094 


0 


562 


-0.017 


Root Mean Square Error 














0.623 



Table 7 

Residuals from Side Length and Rotation Model 



Subject 1 {e " -.40) 



Rotation 



Side Length 




40 




80 




120 




160 


Average 


80 


0 


768 


0 


788 


0 


113 


-0 


163 


0 


376 


100 


-0 


012 


0 


122 


0 


255 


0 


006 


0 


093 


120 


-0 


127 


-0 


471 


0 


654 


-0 


852 


-0 


199 


140 


-C 


072 


-0 


178 


-0 


484 


-0 


591 


-0 


331 


Average 


0 


139 


0 


065 


0 


135 


-0 


400 


-0 


015 



Root Mean Square Error 0.457 



Subject 2 (6 ~ -.32) 



Rotation 



Side Length 




40 




80 




120 




160 


Average 


80 


-0 


934 


0 


373 


-0 


469 


-0 


429 


-0 


365 


100 


1 


339 


1 


040 


-0 


032 


-0 


561 


0 


447 


120 


-0 


696 


-0 


634 


0 


393 


0 


188 


-0 


187 


140 


0 


798 


0 


044 


-0 


395 


-0 


221 


0 


057 


Average 


0 


127 


0 


206 


-0 


126 


-0 


256 


-0 


012 



Root Mean Square Error 0.639 



Subject 3 id - -.40) 



Rotation 



Side Length 




40 




80 




120 




160 


Average 


80 


0 


432 


0 


192 


0 


436 


0 


217 


0.319 


100 


-0 


978 


-1 


237 


0 


397 


0 


619 


-0.300 


120 


-0 


714 


0 


345 


0 


067 


1 


115 


0.203 


140 


-0 


227 


-0 


699 


-0 


517 


0 


304 


-0.285 


Average 


-0 


372 


-0 


350 


0 


096 


0 


564 


-0.016 



Root Mean Square Error 0.626 



ERIC 



4G 



Table 8 

Residuals From Two Strategy Model 



Subject 1 (P^ - .98, 6^ - -.40; P2 - .02, 6^ - -.39) 

Rotation 



Side Length 




40 




80 




120 




160 


Average 


80 


0 


571 


0 


695 


0 


124 


-0 


049 


0 


336 


100 


-0 


188 


0 


048 


0 


285 


0 


140 


0 


071 


120 


-0 


265 


-0 


506 


0 


723 


-0 


680 


-0 


182 


140 


-0 


171 


-0 


174 


-0 


377 


-0 


380 


-0 


276 


Average 


-0 


013 


0 


016 


0 


189 


-0 


242 


-0 


013 



Root Mean Square Error 0.405 



Subject 2 (P^ - .70. - -.32; P2 - .30, 6^ - -.31) 

Rotation 



Side Length 




40 




80 




120 




160 


Average 


80 


-1 


Oil 


0 


338 


-0 


462 


-0 


380 


-0 


379 


100 


1 


268 


1 


Oil 


-0 


019 


-0 


505 


0 


439 


120 


-0 


755 


-0 


651 


0 


419 


0 


256 


-0 


183 


140 


0 


752 


0 


040 


-0 


357 


-0 


141 


0 


074 


Average 


0 


064 


0 


185 


-0 


105 


-0 


192 


-0 


012 



Root Mean Square Error 0.629 



Subject 3 (P^ - .00, - -.41; P2 - 1.00, 6^ - -.40) 

Rotation 



Side Length 




40 




80 




120 




160 


Average 


80 


0 


650 


0 


300 


0 


433 


n 


103 


0 


371 


100 


-0 


786 


-1 


156 


r» 

\- 


367 


0 


479 


-0 


274 


120 


-0 


575 


0 


373 


-0 


015 


0 


921 


0 


176 


140 


-0 


140 


-0 


723 


-0 


652 


0 


058 


-0 


365 


Average 


-0 


213 


-0 


301 


0 


033 


0 


390 


-0 


023 



Root Mean Square Error 0.577 



Captions for Figures 

A Sample Task 

Medi an Standardized Log Response Time versus Rotation 

Median Standardized Log Response Time versus Side Length 

Posterior Probabilities of Use of Rule-Based Strategy 

Posterior Probabilities of Component 1 Membership in a 
Mixture of Two Gaussian Components 

Posterior Probabilities of Use of Rule-Based Strategy, 
Distinguishing Instructed and Noninstructed Subjects 
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Two— Component Gaussian Mixture 




Probability of Component #1 Membership 



0.45 



0.4 - 



0.35 - 



0.3 - 



0.25 -: 



Two Strategy Mode! - Trimmed Data 

Fniqtierxy - post, prior strat 1 



-1 

m 



m 




i 



0,05 0.15 0.25 0.35 0.45 0.55 0.65 0.75 0.85 

fv-^ , Probobility of Using Stro teqy 1 

ISSJ Not Instructed ^ Instructed 



1 



0.95 



Distribution List 



Dr. Terry Ackerman 
Educational Psychology 
210 Education Bldg. 
University of Illinois 
Champaign, IL 61801 

Dr. James Algina 
1403 Norman Hall 
University of Florida 
Gainesville, FL 32605 

Dr. Erling B. Andersen 
Department of Statistics 
Studiestraede 6 
1455 Copenhagen 
DENMARK 

Dr. Ronald Armstrong 
Rutgers University 
Graduate School of Management 
Newark, NJ 07102 

Dr. Eva L. Baker 

UCLA Center for the Study 

of Evaluation 
145 Moore Hall 
University of California 
Los Angeles, CA 90024 

Dr. Laura L. Barnes 
College of Education 
University of Toledo 
2801 W. Bancroft Street 
Toledo, OH 43606 

Dr. William M. Bart 
University of Minnesota 
Dept. of Educ. Psychology 
330 Burton Hall 
178 Pillsbury Dr., S.E. 
Minneapolis, MN 55455 



Dr. Isaac Bejar 
Mail Stop: 10-R 
Educational Testing Service 
Rosedale Road 
Princeton, NJ 08541 

Dr. Menucha Birenbavmi 
School of Education 
Tel Aviv University 
Ramat Aviv 69978 
ISRAEL 

Dr. Arthur S. Blaiwes 
Code N712 

Naval Training Systems Center 
Orlando, FL 32813-7100 

Dr. Bruce Bloxom 

Defense Manpower Data Center 

99 Pacific St. 

Suite 155A 
Monterey, CA 93943-3231 - 

Cdt. Arnold Bohrer 

Sectie Psychologisch Onderzoek 

Rekruterings-En Selectiecentrum 

Kwartier Koningen Astrid 

Bruijnstraat 

1120 Brussels, BELGIUM 

Dr. Robert Breaux 
Code 281 

Naval Training Systems Center 
Oriando, FL 32826-3224 

Dr. Robert Brennan 
American College Testing 

Programs 
P. O. Box 168 
Iowa aty, lA 52243 



Educational Testing Service/Misievy 

Dr. John B. Carroll 
409 EUiott Rd., North 
Chapel Hill, NC 27514 

Dr. John M. Carroll 
IBM Watson Research Center 
User Interface Institute 
P.O. Box 704 

Yorktown Heights, NY 10598 

Dr. Robert M. Carroll 
Chief of Naval Operations 
OP-01B2 

Washington, DC 20350 

Dr. Raymond E. Christal 
UES LAMP Science Advisor 
AFHRVMOEL 
Brooks AFB, TX 78235 

Mr. Hua Hua Chung 
University of Illinois 
Department of Statistics 
101 Illini Hall 
725 South Wright St. 
Champaign, IL 61820 

Dr. Norman Cliff 
Department of Psychology 
Univ. of So. California 
Los Angeles, CA 90089-1061 

Director, Manpower Program 
Center for Naval Analyses 
4401 Ford Avenue 
P.O. Box 16268 
Alexandria, VA 22302-0268 



05/24/90 

Director, 

Manpower Support and 

Readiness Program 
Center for Naval Analysis 
2000 North Beatiregard Street 
Alexandria, VA 22311 

Dr. Stanley Collyer 

Office of Naval Technology 

Code 222 

800 N. Quincy Street 
Arlington, VA 22217-5000 

Dr. Hans F. Crombag 
Faculty of Law 
University of Limburg 
P.O. Box 616 
Maastricht 

The NETHERLANDS 6200 MD 

Ms. Carolyn R. Crone 
Johns Hopkins University 
Department of Psychology 
Charles & 34th Street 
Baltimore, MD 21218 

Dr. Timothy Davey 

American College Testing Program 

P.O. Box 168 

Iowa City, lA 52243 

Dr. C. M. Dayton 
Department of Measurement 

Statistics & Evaluation 
College of Education 
University of Maryland 
College Park, MD 20742 



56 



Educational Testing Service/Mislevy 



05/24/90 



Dr. Ralph J. DeAyala 
Measurement, Statistics, 

and Evaluation 
Benjamin Bldg., Rm. 4112 
University of Maryland 
College Park, MD 20742 

Dr. Lou DiBello 
CERL 

University of Illinois 

103 South Mathews Avenue 

Urbana, IL 61801 

Dr. Dattprasad Divgi 
Center for Naval Analysis 
4401 Ford Avenue 
P.O. Box 16268 
Alexandria, VA 22302-0268 

Dr. Hei-Ki Dong 

Bell Communications Research 

6 Corporate Place 

PYA-1K226 

Piscataway, NJ 08854 

Dr. Fritz Drasgow 
University of Illinois 
Department of Psychology 
603 E. Daniel St. 
Champaign, IL 61820 

Dr. Stephen Dimbar 
224B Lindquist Center 

for Measurement 
University of Iowa 
Iowa aty, lA 52242 

Dr. James A. Earles 

Air Force Human Resources Lab 

Brooks AFB, TX 78235 



Dr. Susan Embretson 
University of Kansas 
Psychology Department 
426 Fraser 
Lawrence, KS 66045 

Dr. George Englehard, Jr. 
Division of Educational Studies 
Emory University 
210 Fishbume Bldg. 
Atlanta, GA 30322 

Dr. Benjamin A. Fairbank 
Operational Technologies Corp. 
5825 Callaghan, Suite 225 
San Antonio, TX 78228 

Dr. P-A, Federico 
Code 51 
NPRDC 

San Diego, CA 92152-6800 

Dr. Leonard Feldt 
Lindqtiist Center 

for Measurement 
University of Iowa 
Iowa aty, lA 52242 

Dr. Richard L. Ferguson 
American College Testing 
P.O. Box 168 
Iowa aty, lA 52243 

Dr. Gerhard Fischer 
Liebiggasse 5/3 
A 1010 Vienna 
AUSTRL\ 

Dr. Myron Fischl 

U.S. Army Headquarters 

DAPE-MRR 

The Pentagon 

Washington, DC 20310-0300 



ERIC 



Educational Testing Service/Mislevy 

Prof. Donald Fitzgerald 
University of New England 
Department of Psychology 
Armidale, New South Wales 2351 
AUSTRALIA 

Mr. Paul Foley 

Navy Personnel R&D Center 

San Diego, CA 92152-6800 

Dr. Alfred R. Fregly 
AFOSR/NL, Bldg. 410 
Boiling AFB, DC 20332-6448 

Dr. Robert D. Gibbons 
Illinois State Psychiatric Inst. 
Rm 529W 

1601 W. Taylor Street 
Chicago, IL 60612 

Dr. Janice Gifford 
University of Massachusetts 
School of Education 
Amherst, MA 01003 

Dr. Drew Gitomer 
Educational Testing Service 
Princeton, NJ 08541 

Dr. Robert Glaser 
Learning Research 

& Development Center 
University of Pittsburgh 
3939 O'Hara Street 
Pittsburgh, PA 15260 

Dr. Bert Green 
Johns Hopkins University 
Department of Psychology 
Charles & 34th Street 
Baltimore, MD 21218 



05/24/90 

Michael Habon 
DORNIER GMBH 
P.O. Box 1420 
D-7990 Friedrichshafen 1 
WEST GERMANY 

Prof. Edward Haertel 
School of Education 
Stanford University 
Stanford, CA 94305 

Dr. Ronald K. Hambleton 
University of Massachusetts 
Laboratory of Psychometric 
and Evaluative Research 
Hills' South, Room 152 
Amherst, MA 01003 

Dr. Delwyn Hamisch 
University of Illinois 
51 Gerty Drive 
Champaign, IL 61820 

Dr. Grant Heiming 
Senior Research Scientist 
Division of Measurement 
Research and Services 
Educational Testing Service 
Princeton, NJ 08541 

Ms. Rebecca Hetter 

Navy Personnel R8tD Center 

Code 63 

San Diego, CA 92152-6800 

Dr. Thomas M. Hirsch 
ACT 

P. O. Box 168 
Iowa aty, lA 52243 



55 



Educational Testing Service/Mislevy 



05/24/90 



Dr. Paul W. Holland 


Dr. Brian Junker 


Educational Testing Service, 21-T 


University of Illinois 


Rosedale Road 


Department of Statistics 


Princeton, NJ 08541 


101 mini Hall 




725 South Wright St. 


Dr. Paid Horst 


Champaign, IL 61820 


677 G Street, #184 




Chula Vista, CA 92010 


Dr. Milton S. Katz 




European Science Coordination 


Dr. Lloyd Humphreys 


Office 


University of Illinois 


U.S. Army Research Institute 


Department of Psychology 


Box 65 


603 East Daniel Street 


FPO New York 09510-1500 


Oiampaign, IL 61820 






Prof. John A. Keats 


Dr. Steven Himka 


Department of Psychology 


3-104 Educ. N. 


University of Nev^castle 


University of Alberta 


N.S.W. 2308 


Edmonton, Alberta 


AUSTRALL\ 


CANADA T6G 2G5 






Dr. Jwa-keun Kim 


Dr. Huynh Huynh 


Department of Psychology 


College of Education 


Middle Tennessee State 


Univ. of South Carolina 


University 


Columbia, SC 29208 


P.O. Bo.i 522 




Murfreesboro, TN 37132 


Dr. Robert Jannarone 




Elec. and Computer Eng. Dept. 


Mr. Soon-Hoon Kim 


University of South Carolina 


Computer-based Education 


Columbia, SC 29208 


Research Laboratory 




Univer sity of Illinois 


Dr. Kumar Joag-dev 


Urbana, IL 61801 


University of Illinois 




Department of Statistics 


Dr. G. Gage Kingsbury 


101 mini Hall 


Portland Public Schools 


725 South Wright Street 


Research and Evaluation Department 


Champaign, IL 61820 


501 North Dixon Street 




P. 0. Box 3107 


Dr. Douglas H. Jones 


Portland, OR 97209-3107 


1280 Woodfem Court 




Toms River, NJ 08753 





Educational Testing Service/Mislevy 

Dr. William Koch 
Box 7246, Meas. and Eval. Ctr. 
University of Texas-Austin 
Austin, TX 78703 

Dr. Richard J. Koubek 
Department of Biomedical 

& Human Factors 
139 Engineering & Math Bldg. 
Wright State University 
Dayton, OH 45435 

Dr. Leonard Kroeker 

Navy Personnel R&D Center 

Code 62 
San Diego, CA 92152-6800 

Dr. Jerry Lehnus 

Defense Manpower Data Center 

Suite 400 

1600 Wilson Blvd 

Rosslyn, VA 22209 

Dr. Thomas Leonard 
University of Wisconsin 
Department of Statistics 
1210 West Dayton Street 
Madison, WI 53705 

Dr. Michael Levine 
Educational Psychology 
210 Education Bldg. 
University of Illinois 
Champaign, IL 61801 

Dr. Charles Lewis 
Educational Testing Service 
Princeton, NJ 08541-0001 



ERIC 

t^Mmmmmm 



05/24/90 

Mr. Rodney Lim 
University of Illinois 
Department of Psychology 
603 E. Daniel St. 
Champaign, IL 61820 

Dr. Robert L. Linn 
Campus Box 249 
University of Colorado 
Boulder, CO 80309-0249 

Dr. Robert Lockman 
Center for Naval Analysis 
4401 Ford Avenue 
P.O. Box 16268 
Alexandria, VA 22302-0268 

Dr. Frederic M. Lord 
Educational Testing Service 
Princeton, NJ 08541 

Dr. Richard Luecht 
ACT 

P. p. Box 168 
Iowa aty, lA 52243 

Dr. George B. Macready 
Department of Measurement 

Statistics & Evaluation 
College of Education 
University of Maryland 
College Park, MD 20742 

Dr. Gary Marco 
Stop 31 -E 

Educational Testing Service 
Princeton, NJ 08451 

Dr. Qessen J. Martin 
Office of Chief of Naval 
Operations (OP 13 F) 
Navy Annex, Room 2832 
Washington, DC 20350 



60 



Educational Testing Servlce/Mislevy 

Dr. James R. McBride 

The Psychological Corporation 

1250 Sixth Avenue 

San Diego, CA 92101 

Dr. Qarence C. McCormick 
HQ, USMEPCOM/MEPCT 
2500 Green Bay Road 
North Chicago, IL 60064 

Mr. Christopher McCvisker 
University of Illinois 
Department of Psychology 
603 E. Daniel St. 
Champaign, IL 61820 

Dr. Robert McKinley 
Educational Testing Service 
Princeton, NJ 08541 

Mr. Alan Mead 
c/o Dr. Michael Levine 
Educational Psychology 
210 Education Bldg. 
University of Illinois 
Champaign, IL 61801 

Dr. Timotiiy Miller 
ACT 

P. O. Box 168 
Iowa aty, lA 52243 

Dr. Robert Mislevy 
Educational Testing Service 
Princeton, NJ 08541 

Dr. William Montague 

NPRDC Code 13 

San Diego, CA 92152-6800 



05/24/90 

Ms. Kathleen Moreno 
Navy Personnel R8tD Center 
Code 62 

San Diego, CA 92152-6800 

Headquarters Marine Corps 
Code MPI-20 
Washington, DC 20380 

Dr. Ratna Nandaktimar 
Educational Studies 
Willard Hall, Room 213E 
University of Delaware 
Newark, DE 19716 

Dr. Harold F. O'Neil, Jr. 
School of Education - WPH 801 
Department of Educational 

Psychology & Technology 
University of Southern California 
Los Angeles, CA 90089-0031 

Dr. James B. Olsen 
WICAT Systems 
1875 South State Street 
Orem, UT 84058 

Dr. Judith Orasanu 
Basic Research Office 
Army Research Institute 
5001 Eisenhower Avenue 
Alexandria, VA 22333 

Dr. Jesse Orlansky 
Institute for Defense Analyses 
1801 N. Beauregard St. 
Alexandria, VA 22311 

Dr. Peter J. Pashley 
Educational Testing Service 
Rosedale Road 
Princeton, NJ 08i:41 



Educational Testing Service/Mislevy 



05/24/90 



Dr. Wayne M. Patience 
American Council on Education 
GED Testing Service, Suite 20 
One Dupont Circle, NW 
Washington, DC 20036 

Dr. James Paulson 
Department of Psychology 
Portland State University 
P.O. Box 751 
Portland, OR 97207 

Dr. Mark D. Reckase 
ACT 

P. O. Box 168 
Iowa aty, lA 52243 

Dr. Malcolm Ree 

AFHRVMOA 

Brooks AFB, TX 78235 

Mr. Steve Reiss 
N660 Elliott Hall 
University of Minnesota 
75 E. River Road 
Minneapolis, MN 55455-0344 

Dr. Carl Ross 
CNET-PDCD 
Building 90 

Great Lakes NTC, IL 60088 
Dr. J. Ryan 

Department of Education 
University of South Carolina 
Columbia, SC 29208 

Dr. Fumiko Samejima 
Department of Psychology 
University of Tennessee 
31 OB Austin Peay Bldg. 
Knoxville, TN 37916-0900 



Mr. Drew Sands 
NPRDC Code 62 
San Diego, CA 92152-6800 

Lowell Schoer 

Psychological & Quantitative 

Foimdations 
College of Education 
University of Iowa 
Iowa aty, lA 52242 

Dr. Mary Schratz 
905 Orchid Way 
Carlsbad, CA 92009 

Dr. Dan Segall 

Navy Personnel R&D Center 

San Diego, CA 92152 

Dr. Robin Shealy 
University of Illiiois 
Department of Statistics 
101 mini HaU 
725 Soutii Wright St. 
Champaign, IL 61820 

Dr. Kazuo Shigemasu 
7-9-24 Kugenuma-Kaigan 
Fujisawa 251 
JAPAN 

Dr. Richard E. Snow 
School of Education 
Stanford University 
Stanford, CA 94305 

Dr. Richard C. Sorensen 
Navy Personnel R&D Center 
San Diego, CA 92152-6800 



6C 



ERIC 



Educational Testing Service/Mislevy 



05/24/90 



Dr. Judy Spray 
ACT 
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San Diego, CA 92152-6800 

Dr. George Wong 
Biostatistics Laboratory 
Memorial Sloan-Kettering 

Cancer Center 
1275 York Avenue 
New York, NY 10021 

Dr. Wallace Wulfeck, III 
Navy Personnel R&D Center 
Code 51 

San Diego, CA 92152-6800 



6-1 



Educational Testing Service/Mislevy 



05/24/90 



Dr. Kentaro Yamamoto 
02-T 

Educational Testing Service 
Rosedale Road 
Princeton, NJ 08541 
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